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The 'Techlash" of the past few years represents a moment of quasi-constitutional upheaval for 
the internet. The way a few private companies have been ''governing" large parts of the digital 
world has suffered a crisis of legitimacy. Calls to find mechanisms to limit the arbitrary 
exercise of power online have gained new urgency. This task of "digital constitutionalism" 
is one of the great projects of the coming decades.^ It is especially pressing in the context 
of content moderation—platforms' practice of designing and enforcing rules for what they 
allow to be posted on their services.^ Historical forms of public and private governance 
offer limited guidance. Platforms are not nation-states.^ But the unprecedented power that 
major tech platforms wield over individual rights and public discourse also differentiates 
them from corporations as we have known them. This outsized impact on rights so central 
to democracy has led to demands for greater accountability for decisions about online 
speech, but extensive government intervention can be a poor tool for achieving this goal. 
Government involvement in speech regulation is uniquely pernicious, and the cure for the 
problems with content moderation should not be worse than the disease. 

Instead, platforms need to innovate to create new forms of self-regulation to meet the 
demands of the moment. To date, the most developed proposal to create an independent 
constraint on its content moderation is that of Facebook's Oversight Board, a court-like body 
that will hear appeals about the most difficult and important content moderation decisions 
Facebook makes and give public reasons for its decisions.^ This essay takes the Oversight 
Board concept as the basis of a possible model in examining the promises and limitations 
of platform self-regulation. While this paper's focus is on the benefits of an independent 
appeals body, to be truly effective any such body needs to be embedded in an entire system 
of governance. As Facebook has itself noted, this is a project that will take years.^ But the 
task is also urgent given its importance and real-world effects.^ 

Semi-independent and transparent self-regulatory oversight mechanisms offer significant 
advantages, not only over the current delegitimized governance structures but also in 
absolute terms. As the actors closest to the front line, platforms will always need to play a 
significant role in drawing lines for online speech, given the high-volume, fast-moving, and 
context-dependent nature of the decisions involved.^ A recent French government report 
acknowledged the benefits of this responsiveness and flexibility in endorsing a model of 
government regulation that "capitaliz[es] on this self-regulatory approach already being 
used by the platforms, by expanding and legitimising it."^ This expansion and legitimacy 
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can come from internal oversight, which can create a forum for the public contestation of 
platform rules and their implementation. But it is also true that self-regulatory solutions are 
likely to be a significant disappointment to many. They will not be able to meet the current 
expansive demands for due process and transparency in most content moderation decisions. 
Nor will they be able to create global norms about the appropriate limits of freedom of 
expression. But these goals set unrealistic benchmarks. 

This essay looks first at the puzzle of why a private company might engage in a kind of 
constitutionalism and the demands that such projects are designed to meet. I then turn to 
the limitations of these measures before explaining their benefits and why they will be an 
important part of the future of online governance. 

Old Governance and New Speech Realities 

This moment demands innovation in platform governance for two main reasons: the growing 
momentum behind the belief that private actors can and should be called on to provide 
greater public accountability for their content moderation practices, and the peculiar needs 
of legitimate speech governance that necessarily shape the form that such accountability 
must take. Together, this creates a need for a new form of governance that can legitimize the 
unprecedented power that private tech platforms have over public discourse. 

Private Public Accountability 

Charges that platforms' content moderation is ''illegitimate" are diverse and hard to 
distill into a simple objection.^ One difficulty is that notions of public accountability and 
constitutionalism initially seem out of place in the context of a private company serving 
willing users. But the major tech companies create speech rules, enforce them, curate and 
censor content, and resolve user disputes, all in the course of moderating some of the most 
important channels of information in the world. A major aspect of the techlash has been 
growing awareness that tech companies lack accountability as they exercise this substantial 
power.^® Their decisions have come to be viewed as inconsistent and arbitrary, and therefore 
illegitimate.^^ It is this demand for public accountability that creates the need for platform 
self-regulatory innovation, even if it is not currently legally required. 

Indeed, platforms have already been systematizing their speech rules and slowly providing 
carefully circumscribed transparency in an effort to garner greater public legitimacy. This 
is not pure public-mindedness. Facebook's head of policy management has written: "Simply 
put, there are business reasons that a big social media company must pay attention to 
what the world thinks of its speech rules."^^ A Facebook-commissioned report by a group of 
independent academics called the Data Transparency Advisory Group (DTAG) explained: 

Facebook has considerable capacity to manage its content regulation process in a top down 
manner which pays minimal attention to users' views. However, the very existence of the 
[Community Standards Enforcement Report] highlights the recognition that public views 
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about Facebook and its attitude about the public matter. They matter for the individual 
user both because disgruntled users find ways to circumvent rules, for example opening 
multiple accounts. In addition, unhappy customers are less likely to use the site and more 
likely to seek alternatives to it.^^ 

But it is one thing to say that, descriptively, platforms have been developing a kind of 
platform law that resembles the common law,^^ or that Facebook's Oversight Board is a kind 
of constitutionalism.^^ It is another thing altogether to suggest that it is actually required 
or normatively desirable to redesign platform governance to create constraints on the way 
platforms exercise their power. This latter idea can seem incoherent to those acculturated in 
American legal norms. 

In the United States, the state action doctrine means that constitutional rights only apply 
to actions taken under the auspices of government.^^ American courts have been reluctant 
to expand the reach of state action beyond narrow bounds,even as the doctrine is 
increasingly strained by more privatization and government outsourcing, use of arbitration 
clauses by companies, and reliance on the private infrastructure that makes much public 
online life possible.^^ The Supreme Court in a June 2019 decision made clear that it has little 
appetite for applying constitutional constraints to tech platforms when it emphasized that 
''merely hosting speech by others is not a traditional, exclusive public function and does not 
alone transform public entities into state actors subject to First Amendment constraints."^^ 
This attitude reflects more than mere constitutional doctrine; it is an underlying ethos 
that focuses on government power as the primary threat to individual liberty. This same 
ethos animated John Perry Barlow's famous Declaration of the Independence of Cyberspace, 
directed to the "Governments of the Industrial World."^® And it reflects a core element of 
the ideal of liberal autonomy that private actors' reasons for acting should be free from 
public scrutiny. From this perspective, a private company's actions do not necessarily need 
public legitimacy beyond what is commercially desirable, and there is no reason to assume 
that the relationship between legitimacy and profitability is linear. 


A 
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But this is not the only way to understand rights. Many other countries give "horizontal 
effect" to rights, reflecting greater recognition of the threats that nongovernmental actors 
pose to rights and a stronger commitment to social democratic norms.The United Nations 
Guiding Principles on Business and Human Rights, a nonbinding but widely endorsed 
statement of international law, also recognizes the obligation of business enterprises to avoid 
infringing on the human rights of others.This "horizontal" model is the intuition behind 
the (mistaken) complaints by users that platforms are infringing their First Amendment rights 
when they moderate. It is also the intuition behind the claims that tech firms possess a kind 
of "sovereignty"^^ or are "public utilities,the equivalent of the modern public square,or 
an old company town.^^ None of these analogies fit easily, and all are attempts to shoehorn 
new problems into more familiar solutions,with potentially detrimental consequences.^® But 
they do underline the problem that self-regulatory innovations aim to meet: a growing sense 
that governments are no longer the only, or even the primary, threat to rights and that when 
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private actors construct ecosystems to manage public rights, these ecosystems should bear the 
characteristics of public accountability, due process, and the rule of law. 

This cultural difference in the understanding of rights means global platforms will face a large 
number of jurisdictions where governments and users are more accustomed to requiring 
private actors to conform to public accountability and rule of law norms.Shifting public 
sentiment, congressional hearings, and even legislative proposals suggest that the tech giants 
will increasingly face this pressure in the United States too. Of course, there continue to be 
practical obstacles to holding corporations accountable for rights infringements in many 
cases.But the trend of the global discourse about tech governance is clear. Platforms have 
an interest in being proactive about the shape of that conversation by creating new forms of 
self-regulation that meet the growing demands for better content moderation performance 
and accountability. 

The need for better systems is exacerbated by laws that increasingly impose legal 
responsibility on platforms for their moderation of public discourse. For example, Germany's 
Network Enforcement Act (NetzDG) delegates the interpretation and enforcement of speech 
regulations to platforms by imposing large fines on platforms that fail to take down certain 
types of illegal content within short time frames.The right to be forgotten in Europe 
represents another example that requires search engines to make judgments about what 
information is in the public interest.These kinds of laws (of which more seem to be on the 
horizon) create greater legal responsibility for company content moderation on controversial 
issues. The concurrent and increasing concern about the legitimacy of these decisions 
by platforms creates the need for a more mature form of content moderation that factors 
accountability and transparency into its design. 

It is worth noting that at present there is nothing legally entrenching platforms' proposed 
self-regulatory reforms. They currently remain voluntary, and platforms may decide to push 
back against the growing demands for public accountability in their content moderation 
ecosystems, reasserting their rights to manage their commercial operations as they please. 
But this may not remain the case. As noted above, France has indicated interest in a regulatory 
model in which increased self-regulation will be mandated for platforms that meet certain 
criteria. But voluntary initiatives by platforms today can shape the form of these future 
mandates. The members of the French mission who endorsed this model did so based on 
'The progress made in the last 12 months by an operator such as Facebook,"^^ showing 
that self-imposed platform reform and regulatory reform can occur in dialogue with one 
another.Whether this is viewed as regulatory capture or a virtuous cycle depends on 
ensuring that the self-regulation that results is effective. 

Speech Is Special (and Especially Difficult) 

There are a number of features of freedom of speech that make designing a legitimate 
system of speech governance especially difficult. Not every feature is unique to speech 
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rights, but combined these considerations show the ways that speech governance raises 
unique challenges. 

First, as Frederick Schauer says, freedom of speech is ''a somewhat different type of bight'" 
because speech is positively advantageous and should be encouraged, not merely shielded 
from state intrusion.This means that regulation of online platforms is different from 
historical forms of gatekeeper regulation, which typically focused on using gatekeeper 
liability to prevent underlying misconduct rather than the promotion of rights. 

The positive benefits of free speech also led to unique doctrinal developments: broader 
standing rights to allow vagueness and overbreadth challenges to statutes and concerns 
about "chilling effects."^® These rules reflect the notion that, more than is the case for 
other rights, any restrictions on speech need to be clearly defined in advance. But this is 
made more difficult because the contours of freedom of speech are essentially contested. 

The frontiers of free speech are especially uneven,^^ differing from country to country.^^ 
Furthermore, what constitutes freedom of expression fundamentally differs in different 
contexts.^^ The fact that context matters so much in translating speech rules into concrete 
decisions in individual cases poses a unique challenge for regulation of speech on global 
platforms, which facilitate large volumes of speech across vastly different settings. As 
Facebook's head of policy management, Monika Bickert, described the problem: 

The practicality of implementing standards in communities this large simply requires a 
heavy hand from the companies. Even if online speech standards were set by an outside 
authority, the level of attention required to implement any standard at such a large scale 
means that companies must play a primary role in the ultimate decision to remove or 
leave on site any given piece of content.^^ 

Free speech cases also involve the collision of different rights more often than other cases. 
Regulating expression, especially hate speech and sexually explicit materials, for example, 
creates conflicts between liberty and equality. One person's liberty of free expression has 
direct impacts on the dignitary interests of others. These clashes are "common and readily 
expressible as 'zero sum' situations."^^ These kinds of zero-sum controversies are especially 
contestable and controversial. Moreover, because they involve balancing different interests, 
they are ill-suited to unilateral, unaccountable, and unexplained decision making. Therefore, 
although freedom of speech is never absolute and can be restricted to take account of other 
sufficiently important interests, these restrictions should be publicly explained and provided 
for in a way that limits discretion.^^ 

But private actors are not well-equipped to evaluate the interests that law typically 
permits freedom of expression to be limited in the name of, such as national security or 
public safety.^^ By contrast, platforms have a legitimate interest in regulating speech in ways 
that would never be permissible for a government. As Tarleton Gillespie put it, content 
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moderation is the commodity that platforms offer.^^ Making choices about speech is their 
business. The way they rank and present content is what attracts or repels users. A website 
focused on knitting should be free to decide it does not want to host certain political 
content that it determines does not align with its mission.^^ This difference causes problems 
in trying to apply existing rules and case law about speech, centered around governments, 
to private platforms that have a wide variety of business models and capacities. 

Perhaps most relevantly (and problematically) for questions of internet governance, state 
involvement in any aspect of speech regulation is especially fraught. First Amendment doctrine 
in particular is deeply shaped by a distrust of government motives in speech regulation.^® 

But other legal traditions have similar concerns. The French report, for example, noted the 
''special precautions" that need to be taken when public authorities become involved with 
speech regulation.^^ Particularly when it comes to regulating political speech, there is a strong 
suspicion that governments might act on illegitimate motives related to preservation of their 
own power. Therefore, state regulation that aims to rein in private power over individual 
expression may itself undermine the democratic purposes of free speech.^^ 

Finally, speech rights are also special because free speech is fundamentally a public-facing 
right. The rights of the individuals involved remain important, but speech is prized for the 
wider benefits it brings to society, and it is sometimes feared for its potential to create wider 
harms. The ramifications of any particular rule or decision for wider public discourse loom 
larger than they do in many other contests over rights. 

To summarize: there is no "correct" understanding of free speech. Restrictions on free speech 
should be clear in advance, but they also need to be applied with special attention to context. 
Who sets speech rules is especially important, and the role of government needs to be carefully 
circumscribed. But outsourcing decisions so central to public discourse to unaccountable 
private actors also causes concern. These are the special challenges of speech governance. 

As a result, regulatory frameworks imported from other contexts cause problems for 
speech governance. For example, multistakeholder initiatives about supply-chain integrity 
may benefit from opaque negotiations if assurances about the protection of commercial 
information make companies more willing to cooperate. But a lack of transparency in 
negotiating respect for freedom of speech is intrinsically inadequate.Another example is 
the increasingly diverse ranges of online dispute resolution (ODR) systems that platforms 
use to resolve other conflicts, such as disputes between a buyer and seller in an online 
marketplace over whether a product sold matched its description. These ODR systems 
typically involve resolving disputes between two private parties on a platform,^^ instead 
of the more "public law" style of disputes that are involved in contesting the ruling of 
the platform "government." While many other ODR systems are increasingly automating 
their processes and removing human handlers from the dispute-management process,^^ 
the nature of speech disputes distinguishes them and makes this automated approach 
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intrinsically flawed.Automated processes struggle with highly contextual decisions, and 
even when algorithms get the decisions ''right," they cannot communicate them in a way 
that will be acceptable to the affected parties. 

Understanding the peculiar problems of speech governance in this way provides insight 
into the role that an internal oversight body, such as Facebook's Oversight Board, could 
fulfill. It also illuminates the most effective role for government regulation. Governments 
cannot and should not micromanage rules for online speech. They can submit permissible 
bounds in a broad sense, in keeping with the restraints of their individual constitutions. But 
for three reasons, these rules will likely not provide an answer in most of the hard cases that 
platforms have to decide in liberal democracies. 

First, platforms will impose restrictions on speech that go beyond what a government might 
be able to, whether for practical reasons (such as to prevent platforms becoming overwhelmed 
by spam or fake accounts) or because of their business model (such as platforms that wish to 
ban adult content, or communities that wish to provide a politics-free zone). 

Second, government regulation could not manage the speed and scale at which platforms 
have to decide these questions. As Bickert notes, online speech regulation requires a 
heavy hand from platforms to translate rules into on-the-ground decisions in a dynamic 
environment. Governments may set broad standards, as in the cases of NetzDG or the right 
to be forgotten, but these only go so far. Online norms of discourse, memes, and coded 
language change by the day, or even the hour. Government processes are unlikely to be 
agile or responsive enough to manage the rapid evolution of online speech disputes, let 
alone apply them to millions of individual cases. 

Third, even if constitutionally permissible or practically possible, the notion of governments 
having such extensive involvement in speech regulation is in deep tension with the 
underlying democratic purposes of free expression. 

Therefore, government regulation of content moderation should focus on legitimizing 
the processes by which platforms make decisions about speech rather than targeting the 
substance of those decisions. Major platforms should be required to show that they are 
moderating in accordance with their publicly stated rules.Platforms may be overtly 
political or politics free, but these values should be transparent to provide accountability. 
Ensuring that content moderation accords with public rules requires platforms to create 
systems that make content moderation more than the sum of individual ad hoc decisions. 

To ensure that platforms moderate speech systematically rather than arbitrarily, governments 
can impose disclosure obligations about the basis of platform decision making, as well as 
require internal oversight mechanisms that render platform decisions publicly accountable. 
These mechanisms can be audited and reviewed for effectiveness; public rules and an 
internal appeals body mean nothing without systems in place to carry out their mandates 
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consistently in other cases. We might call this a model of 'Verified accountability/' 
where platforms have an obligation to make aspects of their governance transparent and 
accountable, while governments regulate to verify these commitments. Another benefit is it 
does not seek to impose one-size-fits-all obligations on platforms, which differ significantly. 
This is not merely a hypothetical idea for regulation: it is the idea behind the French 
report's recommendation of "expanding and legitimising" self-regulation to create greater 
trust and legitimacy for online rules. 

The question then becomes what "public accountability" means in the context of a 
private platform making speech decisions at scale. While internal oversight can improve 
the accountability of the currently mostly opaque systems of content moderation, it is 
important to be up front about the fact that there are a number of demands that these 
mechanisms will not be able to meet. 

The Limits of Self-Regulatory Oversight 

Defining success for self-regulatory content moderation requires realistic expectations and 
an understanding that there are many complaints about platforms' content moderation 
ecosystems that an individual oversight body or a self-regulatory system in general cannot 
address. 

‘"Perfect” Process in Every Case 

One of the most pervasive criticisms of the current way platforms moderate content is that 
their systems do not provide adequate due process to affected users.Many legal systems 
provide an avenue for appeal in order to afford due process because appeals give complainants 
an opportunity to voice their grievances, have hearings, and receive some forms of explanation 
for their treatment.^^ But the scale of the largest platforms makes this expansive conception 
of due process impracticable. As Alex Stamos has noted, Facebook makes more content- 
moderation decisions in one day than the US justice system makes all year.^^ There are also 
difficulties in translating notions of "due process" that center around the exercise of state power 
to private actors. On the one hand, for example, requiring platforms to provide additional 
process does not give rise to the same questions of trade-offs in the use of public resources. On 
the other hand, the use of coercive power by the state to infringe on private rights raises very 
different questions than the exercise of power between two private parties. These differences 
are rarely acknowledged in calls for greater due process in content moderation. 

Nevertheless, there are substantial benefits—including legitimacy—to providing a kind 
of procedural fairness (something akin to what we think of as "due process") online. 

Scale alone does not justify jettisoning these benefits, and platforms are increasingly 
offering greater procedural protections to users. But due process does not mean perfect 
process. Conversations about due process in content moderation should more explicitly 
account for the fact that the right itself varies according to context. This more nuanced 
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discussion around what the '"due" in ''due process" means in the context of the scale of 
online platforms requires grappling with what kinds of errors society prefers. 

Traditionally, US law has erected strong due process protections for speech rights, as against 
governments, requiring procedures used to show "the necessary sensitivity to freedom of 
expression" and acknowledgment of chilling concerns.But the general principle remains 
that due process is flexible and depends on the circumstances.^^ Determining what process 
is due in any case requires consideration of the private interest affected (and, in the context 
of private platforms, the multiple private interests affected—both the platform's and 
the user's), the probable value of any additional procedural safeguards, and the burden 
that such additional safeguards would entail.Critically, "procedural rules must always 
be designed as a system, in light of the overall goal of the programs."^^ Fundamentally, 

"the very nature of the due process inquiry indicates that the fundamental fairness of a 
particular procedure does not turn on the result obtained in any individual case; rather, 
'procedural due process rules are shaped by the risk of error inherent in the truth-finding 
process as applied to the generality of cases.'"^^ 

Discussion of process in the context of content moderation typically does not take 
this systemic view. The Santa Clara Principles developed by civil society organizations, 
for example, call for a meaningful opportunity for timely appeal—which includes the 
opportunity to present additional information and the provision of reasons for decision— 
for "any content removal or account suspension.But Facebook removed 2.2 billion fake 
accounts in the first quarter of 2019 alone, as well as 1.8 billion pieces of content for spam 
violations.^® People appealed 20.8 million of the decisions made in relation to spam. This 
is an average of around 231,000 appeals a day. Facebook takes action against content for 
violating its rules against spam more often than in any other category, but it is still only one 
category in its community standards. Overall, Facebook received nearly 25 million requests 
for appeal of content in the first quarter of 2019. The company does not release information 
on how many additional pieces of content it algorithmically reduced distribution of without 
taking down, an enforcement tool the company is increasingly using to deal with content 
that approaches the line of what it prohibits.Focusing on suspensions should not obscure 
the fact that such down-ranking could have a very similar effect in practice on how many 
people see a post. These numbers make clear that requiring reasons in every case would 
overload even a vastly expanded content moderation workforce. 

It is not clear that a system that allowed an appeal with an opportunity to present 
additional information in all of these cases would serve the goals of greater due process 
in the system overall: there would be trade-offs in consistency between decisions, the 
timeliness of review, and the level of reasoning that could be provided, as well as in 
diverting resources to categories such as spam that could be devoted to other case categories, 
including hate speech and bullying. This is not unique to content moderation; in the 
design of all systems of procedural justice, there can be trade-offs between accuracy in 
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any individual case and overall systemic accuracy/^ It may be that spam is a particular 
category of case where affording fewer procedural rights can be justified—after all, there 
is even government regulation against spam/^ But this just illustrates the point that 
what '"due process" means needs to be determined contextually. It is not enough to say 
that because speech rights are involved, robust due process protections should always be 
provided, particularly when private actors, and not the government, are involved. Of course, 
if different categories of content are treated differently, it will become more important 
to ensure that platforms are accurately categorizing their decisions. This is where self- 
regulatory auditing, enforced and verified by legal mechanisms, can play a key role.^^ 

Explicit discussion has barely begun about how to draw lines between different ways 
platforms treat content, what kind of process should attend each case, and how procedural 
design affects the operation of the content moderation ecosystem as a whole. The UN special 
rapporteur for freedom of expression, David Kaye, for example, has acknowledged the need to 
find "scalable solutions" to the problem of content moderation, including establishing criteria 
for complaints that qualify for additional appeals, given how time-consuming and costly it 
would be to allow appeals on every content action.A more sophisticated conversation about 
due process might start by more explicitly differentiating between the different categories 
of speech that content moderation implicates, accounting for the difficulty of making a 
correct decision in each category, the public importance of the underlying speech, and how 
people experience different kinds of decisions. The stark differences in how people feel about 
content moderation decisions in different categories can be seen in the differences of appeal 
rates. For example, when Facebook finds a piece of content to be hate speech or bullying, 
users appeal those decisions far more often than in other categories.This could reflect the 
more difficult nature of the initial decision, but it also might reflect the greater grievance a 
user feels when they are said to have violated hate speech or bullying policies. Both possible 
reasons should factor in to the design of content moderation systems. 

Another key factor that system design must account for is speed of decision making. Because 
content can go viral online within minutes, reducing the impact of harmful speech often 
requires a platform to remove it quickly. But this inevitably increases the error rate. A 
systematic understanding of due process might recognize that such temporary errors are 
an inevitable and acceptable trade-off against the need to impose rules within a time frame 
that makes them meaningful. Perhaps this means more willingness to accept occasional 
mistakes when educational or journalistic depictions of extremism are temporarily swept up 
in bans on extremist content, for example.Or more consciously deciding that accidental 
censorship of more videos of first-person shooter video games may mean that first-person 
depictions of real-world violent attacks will be more reliably detected.^^ Currently, it is 
difficult to make overt decisions to accept a certain level of error. Instead, as Gillespie 
says, currently "one failure can incur enough public outrage to overshadow a million 
quiet successes."^® But this results in the type of error that systems favor being chosen 
less consciously. Oversight can provide a forum for error explanation and more deliberate 
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choices between trade-offs involved in any system design. However, accepting errors also 
requires trust that companies will still work to reduce them and have systems in place to fix 
them. Again, this is where regulation and auditing can play a role. 

In a dramatically changed speech environment where speech is cheap and attention is 
scarce,^^ and when the biggest threat to free speech may not be censorship but exploiting 
and distorting online discourse through speech itself,the calculus of due process requires 
different conclusions to prior free speech case law that emphasized that the specialness of 
speech required more robust process. Requiring hearings for appeals on all content decisions 
might incentivize laxer substantive moderation rules, which could cause other unintended 
harms. Striking the correct balance between substantive justice, speed, scale, and accuracy 
requires a more conscious calibration of the level of process that is afforded to affected users. 

For this reason, self-regulatory oversight will not, and should not, be an answer to calls for 
expansive due process rights analogous to traditional court hearings about censorship. 

Transparency in Every Case 

Calls for greater transparency have become common in criticisms of content moderation, 
perhaps because transparency is often lauded in policy discourse as low-cost, simple, and 
easy. What is more, company objections can sound defensive and purely self-interested, as 
if there is something to hide. As discussed above, a measure of transparency is essential to 
creating accountability. Certain aspects of content moderation, such as the rules and values 
underlying platform decisions, need to be made public if platforms are to be held to them. 

The problem is that most calls for transparency are generalized and lack specific content. 

But 'Transparency" can mean many things, not all of them equally beneficial, and abstract 
assumptions about its virtue do not advance the conversation about what exactly companies 
should be doing.A simple example is the legitimate need to avoid adverse privacy or 
safety implications of providing too much transparency in cases of hate speech, bullying, 
or reports on dangerous organizations. Republishing and providing greater visibility to 
the content of hate speech or bullying messages as part of an effort at "transparency" 
may in effect do the work of the person the platform has decided should not be heard. 

Similarly, there is the need to avoid the "Streisand effect" in removing "doxing" posts or in 
right to be forgotten cases, where respect for an individual's privacy is the point. Because 
transparency can be an instrumental good for achieving greater accountability, the benefits 
and costs of particular transparency measures should be considered in a more specific 
sense, rather than by reference to abstract concepts such as "legitimacy" alone.What is 
important is, again, a systemic focus that reveals more useful information than a focus on 
public transparency in each individual case.^^ 

While there have been moves toward more transparency in recent years, the enforcement 
of platform rules currently operates largely in the shadows. The major platforms release 
aggregate numbers about enforcement actions taken against various types of content, but 
without examples or further information it is difficult to assess what these reports mean. 

__ m 
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For example, when Facebook says it took down four million pieces of hate speech in the first 
quarter of 2019, more than in any previous quarter, and that this reflects 'Improvements 
and expansion" of their proactive detection methods, it is impossible to assess these 
claims.There is no independent verification of numbers or whether what is being taken 
down actually meets the definition of "hate speech" in Facebook's Community Standards. 

As the French report puts it, "The persistent dissatisfaction of the public authorities can be 
explained in particular by their inability to assess the measurable reality and value of the 
self-regulation carried out by these operators, due to a lack of information validated by a 
trusted third party."^^ Without verification or intelligibility, aggregate reports become a form 
of transparency theater, deployed to ward off calls for greater accountability.^^ Validated 
numbers alone are not enough. Individual examples and sampling of enforcement decisions 
are necessary to illustrate what abstract platform rules mean in practice and the level of 
accuracy of enforcement decisions. 

An independent oversight body can make abstract rules more comprehensible by providing 
examples in a small set of cases. By giving reasons in especially difficult cases, those that sit on 
the borderline of categories, the shape of the category as a whole becomes more understandable. 

But when it comes to development of platform "case law,"^® a single body cannot provide 
transparency in any but the smallest proportion of decisions given the sheer scale of 
platform operations. This is not necessarily a bad thing: privacy and safety concerns will 
often mean individual cases should be kept from public view. In right to be forgotten 
cases, disclosure of identifying information would violate Europe's General Data Protection 
Regulation.^^ But there then needs to be a mechanism for ensuring that in those cases 
platform rules are being applied with an acceptable level of consistency and without bias. 
Internal self-regulation or independent auditing can play a quality-assurance role here. 

An example is Facebook's ongoing civil-rights audit: an independent team of civil-rights 
experts were able to conduct an investigation into the enforcement of Facebook's hate 
speech policies.^^ Auditors were able to observe reviewers at work and examine samples of 
incorrect enforcement decisions. This process resulted in concrete recommendations for 
how Facebook could improve its moderation processes. 

Accepting something less than radical transparency requires trust that there are systems 
in place to ensure fairness in those cases that are not visible. If accountability cannot be 
completely public, it should be verified through system testing. Self-regulatory oversight can 
help remedy the lack of trust created by the history of opacity and obfuscation in content 
moderation to date, but it will not create sweeping transparency in all cases. 

Global Norms 

Platforms often prefer one set of globally applicable content moderation norms, and their 
terms of service or community standards tend to be phrased in these terms. The claim 
is that global rules are necessary for reasons of "efficiency and the completeness of the 
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borderless experience."^^ Platforms no doubt have costs reasons to want to avoid a different 
set of rules for every jurisdiction. But an internal oversight body, even one that prioritizes 
diversity among its members,is unlikely to be able to resolve what are likely intractable 
differences as to appropriate speech norms between different communities. Even if such a 
substantive reconciliation of differences were theoretically possible or normatively desirable, 
it is unlikely that any private company would be able to establish a body with sufficient 
legitimacy or credibility to bring this about.^^ 

Instead, internal oversight bodies should focus not on harmonizing rules between 
communities but on bringing diverse perspectives to bear on how platform rules need to 
be applied differently in different contexts. The history of platform-content moderation 
has partly been one of learning about the need to accommodate local concerns, and the 
realization that US First Amendment norms are not universally applicable or appropriate.^^ 

In hard free speech cases, especially in areas such as political speech or hate speech, context 
is all-important. This means that broad rules—even international human rights norms— 
will have different applications in different situations.^^ While not to understate the 
structural and social harms of racism, the risk of imminent danger created by a racial slur 
may be very different in a Western democracy than in the context of a society with ongoing 
widespread ethnic violence, for example. 

Internal oversight and self-regulation will also be unable to resolve tensions between 
proliferating local laws requiring different rules in different jurisdictions. How and when 
platforms apply these laws is going to be one of the defining questions for the future of 
free speech online. Where government demands are inconsistent with stakeholder values, 
independent oversight might help those stakeholders pressure platforms to push back by 
bringing attention to these cases. David Kaye also suggests that if companies ground refusals 
to accede to government demands in the language of international human-rights law, it 
might increase the credibility and force of those platforms' pushback.^^ Internal oversight 
could facilitate this process if it were charged with explaining how government rules were 
inconsistent with more universal norms. But the role for an oversight body in protecting 
against government overreach is limited. Because government interference in speech is 
considered especially pernicious, this is one area that transparency should be provided as 
a matter of course and not merely in response to individually initiated cases that require 
affected users to bear the burden of raising the issue. And to the extent that governments 
use coercive power to mandate certain content regulation through law, an internal oversight 
body cannot mandate noncompliance. 

What Internal Oversight Can Do 

Despite the limitations described above, internal oversight bodies can be a useful 
innovation and can significantly improve current content moderation ecosystems.The 
exact design of such an institution of course matters a great deal. Facebook's Oversight 
Board is to date the only such proposed initiative, and the publicly available details about 
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how it will operate are still very high level.^®® Furthermore, given the diversity in platforms' 
services and structure, oversight bodies should be tailored to each platform's specific 
context. The discussion that follows is therefore based on an ideal type: an oversight body 
staffed by persons independent of the business arm of the platform that hears appeals 
about the platform's enforcement of its content moderation rules and gives public decisions 
affirming or modifying platform practice that are then implemented throughout the 
platform's content moderation ecosystem. This description obscures a lot of complexity: 
who can bring complaints and how they are chosen from the millions of potential cases are 
key among the details that greatly impact the effectiveness of the institution. Nevertheless, 
this description provides a useful starting point for identifying the advantages that such a 
body can bring, which then can guide specific design decisions. 

These bodies can provide two main benefits: a ''judicial"-style check on platform decisions 
can improve platform policies, and public reasoning can create greater acceptance of rules in 
a pluralistic community where disagreement about the substance of those rules is inevitable. 
An oversight body can also provide greater visibility to community norms, diffuse public 
pressure, and explain moderation decisions about government speech. 

Improve Platform-Policy Processes 

A "judicial"-style check on the "legislative" activities (i.e., the policy formation) of a 
company may improve the overall functioning of its system. Even an oversight body that 
can be overruled can help highlight blind spots and counteract inertia in the formulation 
of content rules.^®^ Blind spots are created both by the fact that policy is often written under 
time constraints and with limited foresight about the full range of circumstances in which 
rules will be applied.^®^ Inertia arises because changing rules is time consuming; in addition, 
when there is persistent disagreement in difficult cases, the status quo often prevails.^®^ 
Judicial-style review counters these problems by creating opportunities to raise issues or 
address complications that arise when policies are applied to specific cases and by providing 
a way to disrupt the status quo. 

The history of content moderation is replete with examples of both these weaknesses. 

Content moderation rules have often been updated haphazardly and in reaction to particular 
scandals.^®^ While public outrage can force an issue, relying on this as a mechanism for policy 
revision favors those in power and makes platforms highly reactive to a "relatively narrow 
spectrum of Western media and politics."^®^ Judicial-style oversight can make retroactive 
review of how policies apply in practice more broadly available, and not just to those who 
can capture public attention. A famous example is Facebook's censorship of the iconic photo 
known informally as "Napalm Girl."^®^ Facebook would not have anticipated this photo being 
swept up by its rules on nudity or child pornography, and when there was public outrage 
over the company's takedown of the historic image, Facebook admitted that it had made a 
mistake and would rewrite its rules to prevent it happening again. But the photo had likely 
been removed thousands of times before the uproar that led to this admission. It was only 
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Beyond identifying weaknesses in policies, an appeals system can help highlight areas 
where the policies themselves are fine but the 'daws on the books" do not accurately reflect 
the rules in practice. In any complex system, enforcement error is inevitable. An appeals 
mechanism acknowledges this reality and formalizes a system for drawing attention to 
mistakes. 

Another benefit of the availability of public review of policies is that it disciplines initial 
policy makers who know that their decisions may later be subject to public scrutiny and 
require justification.^®^ This does not mean opening up all internal deliberations to public 
view; decision makers need a measure of opacity to ensure candor and the ability to consult 
widely.^®^ But it does mean that final decisions are more likely to be made based on publicly 
acceptable grounds because the people who write the rules will be forced to think in 
advance about whether and how their choices can be rationalized. When rules are later 
challenged, policy makers are forced to give their reasons and respond to decisions made 
by an oversight body. By opening up the reasoning process, this public deliberation creates 
legitimacy for rules the community is asked to abide by.^®^ 

Provide Public Reasoning 

Even where self-regulation does not result in substantively different policies, creating a 
forum for public contestation and explanation of rules may give them greater legitimacy. 
"Legitimacy" is a vague concept that can be difficult to define.^^® What is critically 
important for present purposes is that legitimacy does not mean correctness; instead, "in 
circumstances of relatively widespread reasonable disagreement, . . . legitimacy connote [s] 
respect-worthiness."^^^ This framing helps highlight the very low bar of the legitimacy of 
the status quo: because content moderation decisions to date have been highly reactive, 
inconsistent, and poorly justified, they are viewed with little respect by those affected, by 
civil society, and increasingly by the media and the public at large. 


A 

i 


The special nature of speech regulation, where many questions have no "right" answer and 
any outcome will make certain segments of society feel aggrieved, makes public reasoning 
an especially important part of garnering greater legitimacy for these decisions. As John 
Rawls argued, in a pluralistic society where there will always be disagreement about what 
rule is best, the exercise of power over those who disagree with decisions is only legitimized 
through public reasoning that proceeds in a way people might be expected to respect.^^^ 

The goal is not necessarily to create unanimous agreement, but simply to allow for reasoned 
disagreement. Empirical research substantiates Rawls's argument: people's judgments of 
legitimacy do not depend primarily on their obtaining favorable outcomes but are more 
strongly influenced by the processes and procedures authorities use, including whether they 


Hoover Institution • Stanford University 





16 


afford participation, demonstrate impartiality, and show respect for people's interests as 
worthy of consideration.^^^ 

This was the case for David Neiwert, whose Twitter account was suspended when he 
changed his profile picture to the cover of his book about the alt-right, which included 
Ku Klux Klan hoods.^^^ Neiwert thought the suspension was wrong and refused to change 
the picture; the image was about analyzing hate, not promoting it. Representatives from 
Twitter reached out and explained that the company takes a no-tolerance stance on such 
images in profile pictures because they are more prominently displayed on the site. Neiwert 
wrote that the conversation ''was cooperative and [they were] genuinely interested in my 
input. These Twitter officials were able to persuade me, at least, that they very much share 
my concerns."^^^ Neiwert still disagreed with the decision, but once he understood the 
reasoning he agreed to change his profile picture. 

An independent oversight body can more formally facilitate this process. While it may 
only be able to do so in a small fraction of cases, as is the case for apex bodies in other legal 
systems, individual disputes can create opportunities for the articulation of underlying 
rationales for rules that then flow through to their application in other cases. Indeed, this 
appears to be an animating reason for the establishment of Facebook's Oversight Board. 
Facebook already consults with subject-matter experts in the formulation of its community 
standards,^^^ and so it is the public nature of the reasoning process of the Oversight Board 
that is its distinct offering. Giving public reasons creates decisional transparency and 
can facilitate a "global dialogue" between a platform and its users about the impacts and 
justifications of a platform's rules.^^^ This responds to criticisms that platforms' opaque 
decision making interferes with their obligations of clarity, specificity, and predictability.^^^ 
Showing that platforms appreciate the "intellectual, logistic, and moral depth" of content 
moderation decisions makes their conclusions more accessible and comprehensible.^^^ 

This can be made more concrete through the example of right-wing provocateur and 
conspiracy theorist Alex Jones's presence on Facebook. Jones's social-media presence 
has been a source of long-running controversy. He often spreads highly offensive and 
inflammatory content as well as dangerous conspiracy theories. But asking platforms that 
are not democratically accountable to become "arbiters of truth" and engage in drawing hard 
lines about whether commentary is false or acceptable implicates free speech concerns. In 
response to one particular post by Jones, leaked emails show that internal executives were 
having difficulty deciding whether the post violated Facebook's Community Standards.^^® 
One executive referred to the fact that the number of violating comments on the post 
"d[id] not meet the threshold for deletion."^^^ UK executives then pointed to local context, 
noting that the image in question is famous in the United Kingdom, where it is "widely 
acknowledged to be anti-Semitic."^^^ The idea of a "threshold" is not explicit in Facebook's 
Community Standards, and those not familiar with the UK context did not understand the 
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full import of the image. Without public explanation, a decision either way on the basis of 
these considerations would be opaque and hard to evaluate. It would provide little guidance 
for users about what behavior does or does not violate Facebook's policies. Ventilating these 
arguments for and against deletion through public oversight would help legitimize them. The 
response to the eventual removal of Jones from Facebook altogether in May 2019 substantiates 
this. Although many commentators had been calling for his removal for some time, the 
response was mixed. Few defended Jones, but there was frustration with the way Facebook 
executed the ban and the lack of transparency around the reason or timing.^^^ Without public 
reasoning, even those who agreed with the decision thought it was illegitimate. 

It's important to reiterate: public reasoning cannot create substantive agreement with all 
decisions. The matters involved will always be controversial. But because of the public 
significance of speech rules, public reasoning is the only way to make these decisions 
accountable and increase their legitimacy. In the short term, the dividends may not always 
be obvious. When a doctored video of House Speaker Nancy Pelosi, which had been slowed 
down to make her appear drunk, started spreading on social media, Facebook left the video 
up, and a high-level executive appeared on national television to explain the company's 
reasoning. YouTube meanwhile quietly took the video down and gave a dubious rationale 
only when asked.^^^ Yet it was still Facebook, not YouTube, that faced the brunt of public 
outrage. But concerns about legitimacy are concerns about the longer-term functioning of 
the overall system, not only individual cases. A few weeks later, YouTube was enveloped in 
a storm of controversy because of the way it handled complaints from a journalist who was 
being harassed by another YouTube creator. YouTube responded mostly via Twitter in abrupt 
tweets and seemingly kept reversing its position. No one involved was happy with the final 
resolution, and there was public outrage from all sides at YouTube's clumsy handling of the 
matter.^^^ Demands that platforms become publicly accountable for the way they exercise 
their power in matters of public importance are, at heart, calls for public reasoning, and this 
is what an oversight body can provide. 


Develop and Explain Norms 


Another consequential benefit of public reasoning, beyond legitimation, is the greater 
visibility brought to community norms. Through contestation and explanation of these 
norms in a more public forum, more community members may become aware of the rules, 
which in turn helps generate compliance. James Grimmelmann describes the importance 
of this aspect, saying that moderation's "most important mission is to create strong shared 
norms among participants. . . . Moderators can influence norms directly by articulating 
them. They can do this either in general, with codes of conduct and other broad statements 
of rules, or in specific cases by praising good behavior and criticizing bad."^^^ J. Nathan 
Matias similarly found that the visibility of the rules of online communities substantially 
increases compliance and overall participation in the community.^^^ By allowing greater 
visibility and participation in content moderation decisions through public reasoning. 
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internal oversight can embed the process of rule formation in a broader community and 
help norms be formed and tested. 

Diffuse Pressure 

For platforms, an independent oversight mechanism might provide a way to outsource 
controversial decisions.^^^ But this may also be beneficial for other stakeholders. One 
of the reasons why it is especially important who decides what speech is acceptable is 
the fear, and historical actuality, of the use of government or majority power to silence 
unpopular voices. As described above, if governments seek to use their power to silence 
unpopular speech, in most cases a platform oversight body will offer little protection. But 
platform oversight can help protect minorities against majorities that seek to pressure a 
platform directly and not through governments. A meaningfully independent oversight 
body need not be responsive to majoritarian will or commercial pressure. It can therefore 
be an important protection for otherwise marginalized voices in the public sphere. Of 
course, the extent to which this protective aspiration is realized depends on the specifics 
of the institutional design and on ensuring that those charged with oversight are properly 
incentivized and not susceptible to pressure.^^^ 

Explain Government Moderation 

When it comes to the relationship between platforms and governments, the focus has 
generally been on fears that governments will use platforms to achieve speech regulation 
that they themselves could not lawfully do directly. Jack M. Balkin calls this ''collateral 
censorship."^^® As discussed above, collateral censorship is one content moderation concern 
that self-regulatory oversight cannot meaningfully address.^^^ 

By contrast, governments increasingly also find themselves on the other side of content 
moderation. As social media platforms become important forums for governments and 
politicians to communicate with their constituents, questions more frequently arise about 
how platforms should treat content from these actors that violates their rules.^^^ This is one 
of the thorniest issues in content moderation. If platforms lack democratic accountability 
or legitimacy, this concern is strongest when a platform interferes with communications 
between a government and its polity. This might be especially true when the government is 
democratic, but arguably citizens have an interest in knowing the views or actions of their 
rulers even when they did not vote for them. At the very least, it is problematic for a private 
company (and in most jurisdictions, a foreign private company) to decide that the citizens 
have no such entitlement. 

There are no easy ways to make these decisions about how to balance the competing 
interests that arise in these cases—for example, when a politician uses hateful language, 
the prevention of harassment or hate speech and the public interest in relevant information 
come into conflict.^^^ But platform decisions in these cases have, to date, been ad hoc and 
often reflective of real-world power dynamics. 
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For example, the suggestion has been made, only through leaks to the media, that a reason 
Twitter has been slow to remove white-supremacist content from its platform is because 
it might sweep up Republican politicians. This perception delegitimizes Twitter's decision 
not to adopt a removal policy because it becomes viewed as arbitrary or biased.^^^ Similarly, 
it was only through comments to the press that Facebook announced it officially had a 
''state actor" policy that exempted governments from the usual rules in its community 
standards.^^^ This tactic also was to the detriment of the perceived legitimacy of that 
decision. The policy is not reflected in its public-facing standards and is not consistently 
applied; politicians in less important markets (for example, Australia^^^) have been censored, 
while those in the United States have not. The opacity and malleability of these rules is 
problematic in a context where Facebook's content moderation decisions have such direct 
impact on the relationship between people and their governments. 

Clear rules alone are not enough. To deal with some of the above problems, Twitter recently 
announced an updated policy about how it will treat tweets by state actors that breach its 
rules but whose availability might be in the public interest.^^^ The new policy includes a set of 
detailed criteria that will be used to assess whether a particular tweet should be taken down. 

But their application will not always be straightforward: the "immediacy and severity of 
potential harm" of a threat of nuclear war by a government leader, for example, is debatable. 

Twitter's decisions under this policy will, by definition, be political and no doubt contested. 

Therefore, public reasoning is necessary to bring clarity and coherence to these decisions. 

The role of rationalizing and legitimizing how platforms treat government speech is only 
a subset of the role that an oversight body has in moderating content more generally. But 
as platforms are increasingly both the weapon of and a form of exerting power over many 
governments, this is an especially important function. It is also a pertinent reminder of the 
special difficulties inherent in regulating content moderation given that governments' own 
interests are at stake. 

Conclusion 

The intricacies of content moderation require enforcers to be close to the ground but within 
a larger ecosystem that makes enforcement scalable.Governments cannot and should not 
perform this role for speech. Practically, translating broad standards into individual decisions 
would stretch government capacity given the volume at which these platforms operate. 

Normatively, such extensive government interference with speech regulation is inconsistent 
with the underlying purpose of the right to free speech. But current content moderation 
practices are suffering from a crisis of trust. Therefore, platforms need to develop more robust 
internal oversight mechanisms to bring greater legitimacy and coherence to their speech 
decisions. 

Currently, highly consequential decisions about free speech rights are playing out in 

settings that are not well suited to them, whether they be behind closed doors at private //If ^ 
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companies, in the midst of public outrage covered by the media, or in ways that do 
not adequately reflect the very particular relationship that governments should have 
with speech regulation. Individuals and the public interest are poorly represented in 
these conversations. The format does not help foster reasoned disagreement. Internal 
oversight mechanisms, both required and legitimized by government regulation, can 
help create forums that deal with these difficult issues more productively and in ways 
that produce greater accountability. Work on designing the forms of these systems is only 
just beginning. It needs to start by being realistic about what regulation can and can't 
achieve, which allows a more conscious evaluation of trade-offs to take place. A model of 
'Verified accountability" will have shortcomings; it may only be the worst system except 
for all the rest. But it is a beta version of a new form of platform governance that shows 
promise. 
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